Docket No. 80169-0036 (GNX-036) 



Mail No. EL 781400040 US 



Jj Patent Application of 

!5( Zheng Jason Geng 

for 



Method and Apparatus for Omnidirectional Three Dimensional Imaging 



The field of view of a conventional camera or a Ugh P™^ 10 ^ ^ jector . A pair of 
employing a reflective ^^f^ is able to obtain three 

omnidirectional cameras is able to form a unique t M g V combina tion of an 

dimensional images of can also provide a means 

omnidirectional camera and an '« a '! i ^^ l ^^ l JZ d*»* around the camera system 
to obtain quantitative three dunensiona 1 J^™^^ apparatus preS ented herein may offer 
^STS t^t^^ ~- ~ .ewmg angle and 
three dimensional measurement capability. 



-V 



1. Prior Art - Existing Approaches to Large FOV Imaging System 

A number of approaches had been proposed in the past for imaging systems to achieve wide 
field-of-view (FOV). None of them, however, is able to generate 3D omnidirectional images In 
S followmg paragraphs, we give a briefly survey on the state-of-the-art of current imaging systems 
that seek to achieve wide FOV. 

1.1. Conventional Cameras 

Most existing imaging systems employ electronic sensor chips, or still photographic film to 
record optical image collected by its optical lens system. The image projection for most camera 
enses is modeled I a "pin-hole" with a single center of projection. Since sizes of camera lens and 
tne SiaguTg sensor have their practical limitations, the light rays that can be collected by a camera 
2L^d "Lived by the imaging device typically form a con,^ 
Therefore angular field-of-views for conventional cameras are within a range of 5 to 50 degrees. 

JSSJS" camera lens f0r W CCD (Charge C ° Upkd DeV1C6) C P Y " 

angular FOV of 41 .2 degree. 

1.2. Fish-Eye Lenses 

Optical engineers had designed several versions of wide-viewing-angle lens system, called 
the fish-eye lens (see [1],[2]). The fish-eye lens features a very short focal length which, when used 
S plat of co^vLionil camera lens, enables the camera to view object for much wider jmgto 
StooS 180 degree of hemisphere). In general, the wider FOV, the more complicated de ign the 
fisteve lens has To obtain a hemispherical FOV, the fish-eye lens must be quite large in 
Sen^i complex in optical design, and hence expensive Also, it "^JE^KlS 
fish eve lens that ensures single view point constraint, i.e., all incoming principal light rays intersect 
at a sSgle poto to form a fixed viewpoint. This is indeed a problem with commercial fish-eye 
tokS Nikon's Fisheye-Nikkor 8-mm £72.8 lens. The use of fish-eye lenses for wide 
FOV ir^gmg application has been advocated by [3] and [4], among others. Although the _acq^d 
toage^y fish-eye lenses may prove to be good enough for some visualization applications, the 
Sorion compensation issue has not been resolved, and the Wgh unit-cost remain to be major 
todies for its wide-spread applications. The fish-eye lens technique has the advantage of adoptinga 
rta&aUy Positioned camera to acquire a wide angle of view. However the nonlinear property 
taTST semi-spherical optical lens mapping make the resolution along the circular 
botdary of'he image ve£ poor, while the field of view corresponding to the circular boundary of 

m^hnage usually represents a ground or floor where a high resolution of image is required. 

1.3. Multi-Camera System or Rotating Imaging Systems 

Large field of view of objects may be obtained by using multiple cameras in the same 

system eafh poL towards a different direction. However, issues on seamless migration of 

mSSe images is further complicated by the fact that image produced by each camera has d fferent 
objection. The cost for such a system is usually high. The image processing quired by 

Stiple cLiras or rotating camera method to obtain a precise 

of an object takes a long time, which is not suitable for real-time battle field modelmg ana 
reconnaissance applications. 

Another straightforward solution to increasing the FOV of an imaging system is to rotate the 
entire ir^ging systerf about its center of projection (Figure 1). An image sequence acqurred by the 
cameraat different positions are "stitched" together to obtain a panoramic view of the scene. Such 



*n ™ach has been recently proposed by several researchers (see [5], [6] [7]). A very interesting 

^SStX^^i * with a non - frontal toage detector t0 scan *■ world - 

The first disadvantage of any rotating image system is that it requires the use of moving 

ote«te monlS scene wirh mobile objeea. This re*,.* *« use of rotaung sysKms 

static and non-real-time applications. 

In contrast, the invention presented herein, called the omnidirectional 
capturing real-time omnidirectional images without usmg any moving parts, c^dif donal 

. 5 ; mflOPQ with a field-of-view covenng entire hemisphere (18U solid space angie;, 

since it can only provide a wide-angle of FOV at certain time instance, not m all directions. 
i 2 Summary of the Invention 

5 The primary objective of present invention is to provide a set of simple methods and 

! unTquT^S » many practical systems to. need simultaneous 360 degree vrewmg angle and 

= three dimensional measurement capability. 



3 BRIEF DESCRIPTION OF THE DRAWING 

figure 1 shows a conventional way to obtain panoramic images: the composition of multiple views 
of a rotated camera into a panoramic image. 

Figure 2 gives a comparison of the Field of View (FOV) among conventional cameras, panoramic 
cameras, and the proposed omnidirectional camera. 

Figure 3 provides examples of reflective convex mirror for O™^^^^^^* 

impinging location on the mirror 
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Figure 4 Acquire Omni-Directional Image from the OMNI-Mirror: A video camera placed at 
Son C can "see" objects in an entire hemisphere FOV, from a single virtual viewpoint at 

mirror's focal center O. , ^ 

Figure 5 shows an embodiment of the Omnidirectional 3D Camera. 

Figure 6: Circular Variable Wavelength Filter (CVWF). 

Figure 7: Omni -directional Rainbow Light Projection Using the OMNI-Mirror and CVWF 
Figure 8: Omnidirectional Structured Light 3D Camera 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 



4.1. Obtaining Omnidirectional View Using Reflective Mirror 

' To dramatically increase the field of view of an imaging system we propose a Mnwhat 
!"! unusual appri: using a reflective surface (i.g., convex mirror). The field of view of a video 
51 cZmcrEe greatly increased by using reflective surface with properly designed ^ce^pe^ 
1 ^aSdcw SrroHn a car is a daily example of using reflective mirror to increase the FOV of a 
driver. 

i « There are a number of surface profiles that can be used to produce omnidirectional FOV . 

'■" Fieure 3 lists three examples: conic mirror, spherical mirror, and parabohc mirror The optical 
m Stry of to convex mirrors provides a simple and effective means to convert vuleo camera s 
' Q pS^ew into an omnidirectional view around the vertical axis of these mirrors, without usmg 
m any moving part. 

I; At ±s first glance, it appears that the omnidirectional imaging task can be accomplished by 

"sD using J^t ^onvextirror. Unfortunately, this is not the case. ^«^^^?^ A 
□ Soon we know that an image is a two dimensional pattern of brightness (or colors). A 
\'" satisfactory imaging system must preserve two essential characteristics: 

(1) Geometric correspondence: There must be a one-to-one correspondence between pixels in an 

image and point in the scene. 
m Sinele Viewpoint Constraint: Each pixel in the image corresponds to a particular viewing 
m diction defined by a ray from that pixel on image plane through a "pinhole" (single vowing 
point). 

Notice that, although the convex mirrors listed in Figure 3 can greatly increase the FOV and 
mav prove adequate fof certain omnidirectional scene monitoring applications, they are no 
Sfactor^ Paging devices. These reflecting surfaces do not preserve the smgle viewpoint 
consSsW) f F g or a high quality omnidirectional imaging system, all the light rays coming in 
the omni imager head should have a single (virtual) viewing point. 
4.2. Design of the Omni-Mirror That Meets the SVC 



In this section, we will discuss a desirable convex mirror surface profile that satisfies the 
single viewpoint constraint: all the (extensions of) light rays reflected by the mrrror must pass 
through a single (virtual) viewpoint. We call such a reflective mirror the OMNl-mirror. 

Let us first define necessary symbols and terminology. As shown in the Figure V we ^ean 
off-the-shelf video camera with a regular lens whose FOV covers entire surface of the OMM- 
mirror. Since the optical design of camera and lens is nationally symmetric all we need to 
determine is the cross-section function z(r) that defines the mirror surface cross section profile. The 
mirror is then the solid of revolution obtained by sweeping the cross-section about the optical axis. 
^ function of the omni-mirror is to reflect all viewing rays coming from video camera's viewmg 
center (focal point, labeled as C) to the surface of physical objects in the field-of-view. The key 
feature of this reflection is that all such reflected rays must have a projection towards a single 
virtual viewing point at mirror's focal center, labeled as O. to other words the mirror should 
effectively steer Viewing rays such that the camera equivalently sees the objects in the world from a 
single viewpoint O. 

We choose hyperboloid as the desirable shape of the Omni-mirrors. A well-know feature of 
a hyperbolic curve is that: the extension of any ray reflected by the hyperbolic curve onginated 
from one of its focal points passes through its another focal point. If we choose the hyperbohc 
« profile for the OMNI-mirror, and place a video camera at its focal point C as shown ,n Figure 4 the 
Surging system will have a single viewpoint at its another focal point O, as if the video camera 

were placed at the virtual viewing location O. 
j! The mathematical equation that describe the hyperbolic Omni-Mirror surface profile is: 



jl (£±£2l_4=i, where c = 4^^ and/ = 2c. 0) 

• The uniaue feature of the Omni-Mirror is that the extension of the incoming light ray sensed by the 
□ S^SrfSXays passing through a single virtual viewpoint O regardless of the location of the 

projection point M on the mirror surface, 
n 4.3. Acquiring Image Using an Omni Mirror 

S For any point P in the scene, the image reflected from the OMNI mirror to image sensor's 

■ :c= image plane has the radius of d c : 

d c = Yu2 + v* (2) 

where u and v are the pixel indexes on the image plane. As shown in Figure 4, the camera viewing 

angle corresponding to the point on the scene is given by: 

* =tan "'f: o) 

The incoming light ray to the camera can be described by a line equation: 
— / (4) 



tan y c 

Omitting details of mathematical derivation based on the OMNI-Mirror equation (1), we can obtain 
a simple closed-form relationship between the omnidirectional viewing angle a and CCD camera s 
viewing angle 7 C : 



s 



_, 2bc - (b 2 + c 2 )cosy c „. 

<X = tan t—. \P) 

a smy e 

This equation establishes a one-to-one corresponding relationship between a and y e . This 
relationship is important when we perform triangulation calculation in the omnidirectional 3D 
camera system. 

4.4 First Embodiment: Omnidirectional Stereo Camera (OSC) 

With a pair of the omnidirectional cameras, we can easily construct an omnidirectional 
stereo camera. Figure 5 shows a possible configuration of the OSC. Two omni-mirrors are placed 
facel-face with optical axes aligned. The virtual imaging centers, O, and 0 2 , are separated by a 
distance of B, which forms the baseline for the stereo vision. 

The triangulation can be carried out directly from omnidirectional images without the need 
for image conversion. Once a 3D object is detected in the omnidirectional image, the viewing angle 
Yl and 7 2 can be determined from cameras' geometry. Based on equation (5), the virtual viewmg 
angles of this 3D object, a, and a 2 , from the virtual viewpoint O, and O* can be determmed The 
^oKe between one of the viewing center and the 3D object in the scene cane be calculated using 
straight forward triangulation principle: 

iW R _ cos(oQ B (6) 

;f; sinCaj+Oj) 

iJl where R is the Range, i.e. distance between P and O,. 

□ 4.5. Second Embodiment: Omnidirectional Structured Light 3D Camera 
K4.5.1 Projecting Structured Illumination Using the OMNI-Mirror 
% Circular Variable Wavelength Filter (CVWF) 

S Notice that in our Omni 3D camera system, a circular variable wavelength filter (CVWF) is 

used to facilitate the generation of 360° projection illumination through a convex mirror. 

The CVWF is a circular optical glass plate coated with color rings of gradually varying 
wavelengths within visible (400 - 700 nm) or IR ( > 700nm) region. The wavelengths of the 
boated cSfor rings are linearly proportional to their radians measured ^* e rf 2 r ° f " 
glass. This feature provides a simple and elegant way of generating structured light for an entire 
scene without using any moving parts (Figure 6). 

The wavelength of the light X passing through a particular position of the CVWF is a linear 
function of r, the radians measured from the center of the filter glass: 



where X c , X 0 and R are filter parameters: X c is the wavelength corresponding to the filter's 
"center" (lowest wavelength color the filter can generate). ^ 0 is the wavelength corresponding to 
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the filter's "edge" (highest wavelength color the filter can generate). R is the effective radian of the 
filter's "edge". And r is the radians measured from the center of the filter glass. 

If the relative position of the CVWF is fixed with respect to the white light projector, the 
projected light with the same wavelength forms light sheets of circular cordc shapes The cone 
anele G between the light sheet and the normal line of the projector is directly related to the 
wavelength oTthe light. In other word, the wavelength of the light is encoded with tite geometoc 
information of the projected com angle. This fixed wavelength X -to-angle 9 relationship is the key 
idea for our rainbow 3D camera system. 

Omni-directional Rainbow Light Projection Using the OMNI-Mirror 

In order to obtain a Rainbow-like illumination with 360° omnidirectional projecting angle 
we use an Omni-Mirror to reflect the conic-shaped light sheets generated by the white light 
projector and a CVWF. As shown in Figure 7, the projection system has a 360° effective projecting 
angle around the vertical axis of the OMNI mirror. 

The OMNI-Mirror mapping provides an easy and elegant way to determine the projecting 
angle 6 of a conic light sheet based on the projection angle of the white light projector (WLP) y. The 
'□projection line equation is given by: 

i'.lj tan 7 

n '[Combining with the OMNI-Mirror equation 

1,1 ' (z-c) 2 r 2 , , / .2 . t.2 rg\ 

in i - = 1, where c = \u tu \ y J 

r b 2 a 2 

9the reflecting point (r mp , W on the mirror can be calculated by solving a ^^^.^f 
i:n some properties of the OMNI-Mirror and simplification procedures m mathematical manipulations, 
ftthe relationship between the white light projection angle 7, and the omni-projection angle 6 can then 

S"be expressed as: 



, 2bc-{b 2 +c 2 )cosr (10 ) 



0 = tan 



a sin 7 

In other words, knowing the white light projection angle Y and parameters of the OMNI-Mirror, the 
omni-projection angle 0 is fully determined. Since the value of y -determines the projected 
wavelength in the rainbow spectrum X, spatial geometric characteristic s ; of the projected cone , shape 
Rainbow light sheet are fully defined. Although the 3D range calculation of the Omni -3D camera 
does not require the precise relationship between 9 and X, such a concise relationship facilitates a 
simple design, implementation and tests of the omnidirectional rainbow light projector. 

4.5.2. Omnidirectional Structured Light 3D Camera 

Figure 8 shows an embodiment of an omnidirectional structured light 3D camera. An 
omnidirectional rainbow light projector is used herein to produce a spati Tf^J 
illumination in the surrounding scene. An omnid.rect.onal camera is placed co-axially (i.e w til 
optical axes aligned) with the omni-rainbow projector. The virtual projection centers 0„ and the 
E imaging center 0 2 , are separated by a distance of B, which forms the baselme for the 
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triangulation based 3D vision system. The triangulation can be carried out directly from the 
omnidirectional images without the need for image conversion. Once a 3D object is detected m the 
omnidirectional image, the viewing angle y 2 is determined from cameras' geometry. The projection 
angle Yl is determined from the wavelength of the structured light projection, due to the .one-to-one 
corresponding relationship of the Rainbow projector between the wavelength and the Projection 
angle The distance between the virtual imaging center Q, and the surface points on the 3D object 
in the scene can be calculated using straight forward triangulation principle: 

R _ coste) B (11). 
sin^+a,) 
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